Repository navigation
[Kimi-K3][LMCache] LMCache offload for Kimi-K3 - #2053
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
CI's reviewdog run on #2053 flagged five findings, all introduced by this branch. Every one is real and still present. atom/kv_transfer/offload/hybrid/dsv4/connector.py:27 I001 atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:11 I001 atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:425 SIM102 tests/test_scheduler.py:1466 I001 tests/test_scheduler.py:1498 I001 plus one more the bot has not reported yet because reviewdog filters to diff context and that line has not changed since: atom/model_engine/page_unit_checkpoint.py:6 I001 --- why the local checks said clean `ruff check` with no `--select` runs the default rule set (E4/E7/E9/F), and neither `I` (isort) nor `SIM` is in it. There is no `[tool.ruff]` in `pyproject.toml` and no `ruff.toml`, so every local run in this branch's history measured the default set and reported "36, unchanged" against the base. That number was true and I reported it as if it meant CI would be clean, which it does not. Checked from now on with `--select I,SIM` as well. --- scope `--fix` over the whole tree repairs 100 findings in files this branch never touched; that was reverted. Fixed here are only the findings this branch introduced, established per file by running the same rules against the same file on the base: for f in $(git diff --name-only <base> HEAD -- '*.py'); do compare ruff --select I,SIM on <base>:$f against $f done Six files come back with more findings than their base; the six above are them. After this commit that comparison returns nothing. The 14 findings that remain across files this branch touches are all pre-existing on untouched lines, which is also why reviewdog does not report them. SIM102 is `should_defer_free`'s stalled-save escape: two nested `if`s collapsed into one condition, same short-circuit order, no behaviour change. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 3936 passed, zero failures. black clean; ruff default set 36, unchanged; ruff `--select I,SIM` shows zero findings introduced by this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1fe7ce5 to
76a10fa
Compare
CI's reviewdog run on #2053 flagged five findings, all introduced by this branch. Every one is real and still present. atom/kv_transfer/offload/hybrid/dsv4/connector.py:27 I001 atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:11 I001 atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:425 SIM102 tests/test_scheduler.py:1466 I001 tests/test_scheduler.py:1498 I001 plus one more the bot has not reported yet because reviewdog filters to diff context and that line has not changed since: atom/model_engine/page_unit_checkpoint.py:6 I001 --- why the local checks said clean `ruff check` with no `--select` runs the default rule set (E4/E7/E9/F), and neither `I` (isort) nor `SIM` is in it. There is no `[tool.ruff]` in `pyproject.toml` and no `ruff.toml`, so every local run in this branch's history measured the default set and reported "36, unchanged" against the base. That number was true and I reported it as if it meant CI would be clean, which it does not. Checked from now on with `--select I,SIM` as well. --- scope `--fix` over the whole tree repairs 100 findings in files this branch never touched; that was reverted. Fixed here are only the findings this branch introduced, established per file by running the same rules against the same file on the base: for f in $(git diff --name-only <base> HEAD -- '*.py'); do compare ruff --select I,SIM on <base>:$f against $f done Six files come back with more findings than their base; the six above are them. After this commit that comparison returns nothing. The 14 findings that remain across files this branch touches are all pre-existing on untouched lines, which is also why reviewdog does not report them. SIM102 is `should_defer_free`'s stalled-save escape: two nested `if`s collapsed into one condition, same short-circuit order, no behaviour change. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors) -> 3936 passed, zero failures. black clean; ruff default set 36, unchanged; ruff `--select I,SIM` shows zero findings introduced by this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds Kimi-K3 support to the lmcache_offload connector by introducing a CPU offload tier for K3’s per-request recurrent (KDA) state and making KV+state resumes joint (both legs must agree on a single safe boundary). This extends the existing dense offload path with K3-specific state spill/load plumbing, new engine-side bookkeeping, and substantial test coverage.
Changes:
- Introduce
kimi_k3offload layout: worker-side state tier (store/load), scheduler-side joint-boundary policy, and shared-pool keying with build-safety (layout_id) separation. - Add engine-side
StateOffloadIndex+ BlockManager/Scheduler wiring to enqueue state loads/stores, settle reports, and expose new cache stats counters. - Register per-request state tensors via
KVCacheTensor.per_request_stateand exclude them from block-addressed KV movers; add backend hooks (state_entry_views,page_unit_views) to name state bytes for packing.
Reviewed changes
Copilot reviewed 39 out of 39 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_state_checkpoint.py | Adds extensive tests for tier load behavior and joint boundary semantics. |
| tests/test_scheduler.py | Tests offload-resume accounting and stalled deferred-save reclamation behavior. |
| tests/test_page_unit_checkpoint.py | Tests PAGE checkpoint nomination/pinning/offload vote behavior. |
| tests/test_multi_connector.py | Ensures multi-connector routes state loads and exposes _state_tier. |
| tests/test_lmcache_offload_v4_page_slot.py | Updates tests to use shared max_pending_saves helper. |
| tests/test_lmcache_offload_connector.py | Adds K3-specific scheduler contract tests (joint boundary, state channels, shell forwarding). |
| tests/test_kda_checkpoint_slot_copy.py | Adds coverage for page_unit_views correctness and image trimming. |
| tests/test_block_manager.py | Tests widened claim behavior + state-tier index installation gating. |
| docs/environment_variables.md | Documents offload connector behavior and layout selection; introduces a new section. |
| atom/plugin/sglang/attention_backend/attention_gdn.py | Marks recurrent per-request state tensors as per_request_state=True. |
| atom/plugin/rtpllm/utils/forward_context.py | Same per_request_state marking for RTP plugin path. |
| atom/model_ops/attentions/kimi_mla_gdn_attn.py | Implements page_unit_views + marks per-request state tensors. |
| atom/model_ops/attentions/gdn_attn.py | Adds state_entry_views and marks recurrent state tensors as per-request state. |
| atom/model_ops/attentions/deepseek_v4_attn.py | Adds state_entry_views and clarifies state-copy docstring/comments. |
| atom/model_ops/attentions/backends.py | Defines new state_entry_views abstract hook for backends owning per-request state. |
| atom/model_engine/state_offload.py | New engine-side state offload index + connector-host gating helper. |
| atom/model_engine/sequence.py | Adds fields to track state load intent and joint-boundary extents. |
| atom/model_engine/page_unit_checkpoint.py | Adds offload nomination/pinning lifecycle + vote through CPU-tier reachability. |
| atom/model_engine/model_runner.py | Publishes attention backend to transfer tensors so tier can access state_entry_views. |
| atom/model_engine/llm_engine.py | Exposes new joint/tier counters in cache statistics. |
| atom/model_engine/engine_core.py | Adds worker termination hardening + periodic stalled-save reconciliation + dispatch of state load/store publishing. |
| atom/model_engine/block_manager.py | Implements joint boundary selection + widened claim + state-tier load/store APIs and funnel aggregation changes. |
| atom/kv_transfer/offload/metadata.py | Adds state_loads / state_stores to offload metadata and work-field declaration. |
| atom/kv_transfer/offload/hybrid/kimi_k3/state_tier.py | New worker-side tier executor for state store/load plus joint-park helper. |
| atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py | New codec for state objects (stable keying + packing/unpacking). |
| atom/kv_transfer/offload/hybrid/kimi_k3/staging.py | New bounded staging helper for whole-entry pack/unpack. |
| atom/kv_transfer/offload/hybrid/kimi_k3/connector.py | New K3-specific connector/scheduler implementation (state tier + joint KV clamp + stall guards). |
| atom/kv_transfer/offload/hybrid/kimi_k3/init.py | New package marker for K3 offload layout. |
| atom/kv_transfer/offload/hybrid/dsv4/connector.py | Uses shared max_pending_saves + routes claim computation through _claim_after_load. |
| atom/kv_transfer/offload/dense/kv_byte_codec.py | Skips per_request_state tensors in dense KV packing. |
| atom/kv_transfer/offload/dense/connector.py | Adds _may_emit_save seam + uses _claim_after_load for post-load claim. |
| atom/kv_transfer/offload/connector.py | Adds kimi_k3 variant selection and strengthens shell docs/forwarding. |
| atom/kv_transfer/offload/config.py | Adds kimi_k3 alias and selection rule based on HF model_type. |
| atom/kv_transfer/offload/atom_lmcache_staging.py | Makes _env_flag robust to whitespace/empty values. |
| atom/kv_transfer/offload/_offload_common.py | Adds _claim_after_load seam + factors max_pending_saves helper. |
| atom/kv_transfer/disaggregation/types.py | Adds ConnectorMetadata.WORK_FIELDS + has_work() and uses it in work detection. |
| atom/kv_transfer/disaggregation/multi/multi_connector.py | Delegates has_work() to subs, re-exports _state_tier, and routes state loads. |
| atom/config.py | Adds KVCacheTensor.per_request_state flag to separate per-request state from block-addressed KV. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def __init__(self, config) -> None: | ||
| super().__init__(config) | ||
| self._state_tier = None | ||
| # Inert until a request has both legs, which only a joint boundary | ||
| # produces; costs one dict lookup per report otherwise. | ||
| self._joint_park = _JointPark() | ||
|
|
| def get_finished(self): | ||
| out = super().get_finished() | ||
| if self._state_tier is None: | ||
| return out | ||
| indexed, index_failed = self._state_tier.take_store_reports() |
| | Variable | Type | Default | Description | | ||
| |----------|------|---------|-------------| |
Split out of PR #1960 so they can land on their own. None of them is about state offload, and none needs #2045; blocking them behind either is pure loss, and pulling them out narrows what is left to review by a few hundred lines. --- a worker that outlives the join keeps the node `EngineCore.exit` joined each worker with a 5s timeout and did nothing after it. A worker that outlives that join keeps its VRAM slice and its all-reduce IPC handles, and `multiprocessing`'s atexit handler then joins the same process again with NO timeout -- so an engine already on its way out parks in `_exit_function -> join` forever while /metrics keeps answering 200. Seen twice on the k3-dev line; only a manual kill recovered the node. `enable_orphan_reaping` cannot help: `PR_SET_PDEATHSIG` fires when the parent *dies*, and this parent never does. Now escalates terminate -> kill. --- a stalled offload save hung the engine with every GPU idle A deferred save's blocks are freed only on `finished_saving`, but LMCache's pin monitor force-unpins a stalled transfer after `pin_timeout_sec` WITHOUT emitting that report. `has_pending_kv_work()` then never clears and the engine busy-loops with every GPU idle -- a hard hang under a tight pool, a slow block leak under a large one. `Scheduler._reconcile_stalled_deferred_saves`, wired into the KV progress poll, reclaims a save deferred past an abandon window and self-throttles to 5s so a 1ms poll costs nothing. The window reads LMCache's own `LMCACHE_EC_PIN_TIMEOUT_SEC` and adds 30s rather than taking an ATOM knob of its own, because that ordering IS the safety argument: two independent env vars would let ours sit below the timeout it must exceed, and nothing would say so. Unset gives 330s. Non-positive disables reclamation and restores the wait-forever behaviour. What makes reclaiming safe is not the clock -- our clock starts when the free is deferred, LMCache's when it pins, and nothing aligns them. It is that `OffloadWorkerMixin._guard` reports `finished_saving` on both the success and the exception path, so a report is lost only when `store()` neither returns nor raises. Two cases follow and they are exhaustive: the parked save is not copying (LMCache force-unpinned its source and stopped reading), and a save queued behind a parked one never reached `store()` at all. `finished_sending`'s `assert seq is not None` becomes a `continue`: after a reclaim, a late completion report has nothing left to free. --- a state-only step under `multi` parked its request forever `connector_metadata_has_work` decides whether a step's metadata reaches the worker, and it decided from a hardcoded field list shared by three connector families. `MultiConnectorMetadata` holds none of the work itself -- everything is in `metas` -- so it mirrored the subs' fields as aggregating properties, and missed one. Every step whose only work was in an unmirrored field was discarded, and the request parked on it waited for a report nobody was asked to produce. Fixed by asking the metadata instead of inspecting it: each class declares its own `WORK_FIELDS` next to the fields it defines, `has_work()` accumulates down the hierarchy, and `MultiConnectorMetadata` delegates to its subs rather than mirroring them -- so a field added to a sub can no longer be missed here. The old union list stays only as the fallback for duck-typed doubles, marked as not the place to register anything new. --- two behaviour-neutral seams and one marker `OffloadSchedulerMixin._claim_after_load` returns `max(hbm, lmc)`, which is what all four inlined call sites did. A seam rather than four `max`es because a layout whose claim and transfer aim at different boundaries has to override it, and getting that wrong is silent: the request starts further along than its state supports and the output is wrong with no exception. `DenseOffloadScheduler._may_emit_save` returns True. It matters for a layout whose `should_defer_free` pins a finished request's blocks until its save drains -- an unbounded queue then lets a slow backend hold an unbounded slice of the pool. `max_pending_saves` moves to `_offload_common` so every layout reads the same `OFFLOAD_MAX_PENDING_SAVES`. `KVCacheTensor.per_request_state` marks a hybrid's slot-addressed recurrent state. It must be registered in `kv_cache_data` -- the linear-attention forward reads it from there -- but no block-addressed mover may touch it, and `DenseKVByteCodec` now skips it: including it either fails the divisibility check or, if the slot count happens to divide `num_blocks`, inflates `bytes_per_block` past what `block_regions` describes. Verified on top of #2045 (`4df58434`): pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3871 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 1 of remounting PR #1960 on #2045. Deliberately a RENAME, not a redesign: the tier still hangs off `StateSlotPool` and still spills on eviction through the staging ring. Phase 2 deletes the ring; Phase 3 moves the mount point to `PageUnitCheckpointStore`. Keeping those apart is what lets a reviewer tell rebase noise from design decisions. Squashed rather than replayed for the reason #2045 gives for its own squash: other's resolutions. #1960 keeps the original history. --- what #2045 renamed underneath the branch StateGroupPool -> StateSlotPool lookup_group(h) -> lookup(h) seq.per_req_cache_group -> seq.state_slots (list) + seq.state_slot num_per_req_cache_groups -> num_state_slots + state_slots_per_req pop() / release() -> pop_many(width) / release_many() has_free() -> has_free(n) _attach_state_group() -> _attach_state_slots() The semantic change hides in that table: **the "group" concept is gone.** Slots are allocated one at a time, `pop_many` states outright that the set is not adjacent and nothing may assume it is, and a checkpoint occupies exactly ONE slot -- speculation scratch is not state anybody resumes into. Every `group * span` address computation in #1960 lost its basis, so the tier's `_spill` / `_resumable_from` / `has_pending_spill` / `take_spill_copies` are re-expressed over slots, and `(group, staging)` becomes `(slot, staging)`. --- the three places the two PRs actually collide `_attach_state_slots` now returns bool. #2045 rewrote its body for per-slot allocation; #1960 needs it to be able to say "the state behind this boundary is not really here", because with the tier voting a hash can be accepted by `_resumable_from` and still not be in HBM. The two compose: #2045's width logic runs, and only the committed slot (element 0) is the load's destination, so only it takes `pop(spill=False)` -- the rollback scratch is this request's own and has nothing to forgo. `allocate` keeps #1960's two extents (`num_cached_blocks` vs `claim_blocks`) on top of #2045's per-slot attach, and the fresh-block loop starts from `len(seq.block_table)` rather than `num_cached_blocks`, since the widened claim already put blocks there. `CacheStats` gains #1960's `total_offload_tokens` alongside #2045's `total_reusable_tokens`. They are not the same series and neither subsumes the other: `reusable` is the honest denominator for the HBM walk, while tier tokens sit ABOVE that walk by design, so folding them in would make `cached > wanted`. `update()` therefore takes five required arguments and one defaulted. `deallocate` keeps #2045's midstep cancellation and `release_many`, and #1960's in-flight-load hold: the committed slot is parked in `_orphan_load_slots` until `settle_state_load`, because a worker is writing it; the scratch goes back immediately. `checkpoint_funnel` reports through `state_checkpoint_fates()` (a sum across `state_caches`) rather than the single `_state_checkpoint_cache` call. Today `state_caches` is a 1-tuple and the two are equivalent -- kept so Phase 3's tier counters, which arrive from a second class, are not silently dropped. --- functionality is NOT verified by this commit Three gates all take the else branch for K3 under #2045, so the tier is inert: 1. `isinstance(cache, StateSlotPool)` -- K3's `state_caches` is `(PagedStateCheckpointCoordinator,)`, so `cache.offload` is never set. 2. `_joint_kv_boundary` returns `_no_joint("not_hybrid")` whenever `paged_state_checkpoints is not None`, which for K3 is always. 3. `_attach_state_slots` returns from its PAGE branch before reaching `_tier_can_serve`. That is deliberate and is what Phase 3/5 undo. Do not read the green suite below as evidence the feature works -- the exit criterion for the tier is a nonzero `state_tier` counter in the logs, which no unit test can produce. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3915 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…to rescue Phase 2. About 1000 lines go, and none of them for tidiness: the ring existed to solve exactly one problem, and #2045 removed the problem. --- why the ring existed at all A checkpoint used to live in a state slot, and `StateSlotPool.pop` handed that slot to the next request the instant it was evicted. The bytes therefore had to be rescued somewhere before they were overwritten, which is what the ring was: K entries allocated inside the state arena, never leased to a request, and addressed as `num_slots + staging` so `state_entry_views` could reach them with no second scheme. Everything else -- `staging_entries`, `admission_entries`, the drain protocol, the starvation detector, the reclaimer, `pop(spill=)`, `spills_forgone`, the spill-before-relocate ordering in `CommonAttentionBuilder.build` -- was a consequence. Under #2045 a K3 checkpoint is a PAGE image in the KV pool whose 127 units are *reserved*, and reserved units are structurally un-takeable: `_take_free` draws only from `_free` and they sit in `_used`, `BlockPool.free` raises on one, and `retire_top` refuses outright (`if top in self._raw_unit_owner: return None`). The source holds still on its own, so a transfer needs no rescue -- exactly the property dense KV has always had, where `should_defer_free` alone carries a save through its D2H and no ring was ever needed. So this is not a simplification, it is the removal of a workaround whose premise is gone. `state_pool.py` and `sub_pool_spec.py` now have an EMPTY diff against --- what stays, and why `state_entry_views` stays (re-indexed from group to slot, since a checkpoint is one slot wide now). Phase 4's load writes the committed slot directly, so this is its destination -- the plan's earlier draft had it deleted, which was wrong. `StagedTransfer` / `_StagingBuffer` stays. It shares a word with the ring and is a different thing: a per-thread pinned host bounce buffer that D2H and H2D both need, sized in bytes and unrelated to slot count. `StateOffloadIndex` keeps its load half (`request_load` / `complete_load` / `fail_load` / `abandon_load`) and its optimistic hash set. `confirm_spill` becomes `note_stored`, which is what it always did. `StateOffloadTier.submit_spill` becomes `submit_store(h, slot)` and loses its `ready_event`: there is no compute-stream producer to fence against once the packer gathers straight from a source that is not moving. --- what this leaves inert, deliberately **Nothing calls `submit_store` after this commit.** Phase 2 removed the ring that used to drive it and Phase 3 gives it its new caller, whose source is the checkpoint's PAGE units rather than a slot. Nothing populates `StateOffloadIndex.hashes` in between, so `_tier_can_serve` is false and no load is ever offered -- the tier is fully inert, on top of the three gates Phase 1 already documented. That is a WIP state, not a shippable one. --- one dead-code find, pre-existing on #1960 A method between `_tier_can_serve` and `_request_state_load` had lost its `def` line in an earlier rebase, leaving its docstring and `return` as unreachable statements after `_tier_can_serve`'s own return. It reads like a live invariant ("Whether `hit_hash`'s state can be produced at all, from either tier") and has no caller: the tier's vote actually reached `_gated_hit` through `StateSlotPool._resumable_from`. Removed rather than restored, since Phase 4 puts the vote on the coordinator side. Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3906 passed, 326 skipped, 3 xfailed, zero failures. black clean; ruff 36, the same count as #2045. `git diff` against Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 3a. Deletes the tier's separate `atom-state-{rank}` LMCache engine, its
`OFFLOAD_STATE_CPU_SIZE` knob, and its `cache_policy = "FIFO"` override. The
codec now binds the same `storage_manager` the KV leg uses.
--- what changed
- `_state_storage_manager()` is gone (~50 lines). `bind_storage_manager` is
handed `self._engine.storage_manager` directly. The `gib <= 0` branch that
already did this was the whole implementation; what goes is the `> 0` half.
- `OFFLOAD_STATE_CPU_SIZE` removed, from the code and from
`docs/environment_variables.md`.
- `cache_policy = "FIFO"` removed; state inherits the KV pool's LRU.
--- why the two-pool argument does not hold
It rested on "the KV write stream is several times the state volume, so it
evicts the checkpoints". Measured against the code rather than assumed:
`page_unit_bytes = num_MLA_layers * block_size * entry * itemsize = 442,368`
with `--block-size 128` pinned by the recipe, so KV costs 3,456 B/token and a
53.6 MiB state image is worth ~16k tokens of it -- 13% of a 117k-token prompt's
KV. (Read the MLA layer count the other way, as a quarter of 93 layers, and it
is 3.3%. The conclusion is the same either way, which is why it did not need
settling first.)
And the byte ratio is not what decides an LRU eviction anyway. A request writes
its ~457 KV chunks and its one state object inside the same prefill window, so
both enter the LRU at the same position and cool at the same rate. State is
retired ALONGSIDE its own KV -- which is the behaviour we want, because a
boundary whose KV is gone is worthless.
Two pools cost what one does not: two eviction policies that must drift, while
a joint boundary needs both legs to survive together. #2045 made this same
argument one tier up -- state moved out of its reserved slots into the KV pool,
81.85% -> 93.60% -- and `OFFLOAD_STATE_CPU_SIZE` is the same reserved capacity
one tier down, so keeping it repeats the thing #2045 disproved.
`LMCACHE_MAX_LOCAL_CPU_SIZE` is now the one size to tune.
--- the cost, stated rather than hidden
`cache_policy` is per-engine, so one pool means one policy for both legs. That
is a real loss of freedom and it is currently free, because LRU is right for
both. The FIFO override's premise was "a state entry is written once and read
once"; #2045's own numbers refute it (~4,808 resumes over ~1,508 checkpoints is
~3 reads each), and re-access is bursty -- branches and retries on one boundary,
then superseded -- which is LRU's home ground. The genuine want, if it shows up,
is "stickier than LRU gives it", which FIFO does not provide either; the lever
then is touching the state key on a KV hit, not splitting the pool back apart.
Nothing yet measures whether that is needed. The counter to add with the store
path (Phase 3d) is `joint_boundary_state_miss`: a boundary whose KV leg landed
and whose state leg missed. Zero means the two pools never needed separating.
Verified: pytest tests/test_lmcache_offload_connector.py -> 186 passed. black
clean; ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 3b. `StateByteCodec.key()` stops being the bare ATOM hash; it becomes
`xxh64(hash || layout_id)`. `get()` gains a size check on what comes back.
--- what changed
- `StateByteCodec.__init__` takes a required `layout_id` and refuses an empty
one. `_build_state_tier` reads it off `state_runtime.checkpoint_spec` and
declines to build the tier at all when it is absent, rather than falling
back to an unqualified key.
- `key(h)` hashes `h` and the layout id together with `xxh64`.
- `get(h, ...)` measures the returned `MemoryObj` and, on a size other than
`entry_bytes`, discharges its reference, counts `_misfit_reads`, warns, and
returns a miss.
- `_object_bytes()` reads `get_size()` with the tensor as fallback, and
returns None when neither answers.
--- why the key needed both
*Build safety.* The same prefix hash names a completely different image under a
different `num_spec`, TP size, or conv/ssm dtype. A size mismatch throws at
`entry_bytes`, but **the same size with a different order or meaning reads back
silently wrong state** -- and a request resumed onto wrong state produces wrong
output with nothing raised anywhere. #2045 already encodes all of it in
`layout_id` (layers / conv shape+dtype / ssm shape+dtype / order / tp / spec /
carry) and enforces it HBM-side in `_validate_paged_state_op`. This is the CPU
side of the same check, and it was missing.
*Namespace separation.* `CacheEngineKey` is
`(model_name, world_size, worker_id, chunk_hash, dtype)` -- no field says what an
entry IS. KV chunk keys carry `ChunkedTokenDatabase`'s chunk hash and state keys
carry ATOM's chained block hash: two different hash functions writing plain
integers into one space. Until Phase 3a they at least lived in different pools;
now they share one, and the only thing keeping them apart was accidental -- this
side hard-codes `torch.uint8` while KV carries the KV dtype. `MemoryFormat` is
not available as a discriminator either: the tier's own comment records that its
`KV_2LTD` is inert, chosen only to pass the allocator's `raise ValueError`.
Folding the layout in makes the two spaces disjoint by construction, since no KV
key can carry a K3 layout id.
`xxh64` rather than Python's `hash((h, layout_id))`: `hash` of a str is salted
per process, so a restart would silently orphan every entry the previous run
wrote -- a cache that quietly starts cold and never says so. `xxh64` is what the
block hashes themselves use.
--- why the size check, if the key already prevents it
It is unreachable, and that is the point: reaching it means two things collided
in the shared pool, and unpacking a colliding object writes another entry's
bytes over a request's live state. Degrading to a miss costs one recompute; not
checking costs silent wrong output. `_misfit_reads` is the counter that says the
key stopped doing its job.
None from `_object_bytes` means "cannot measure", never "size 0" -- refusing
what cannot be measured would turn an unknown into a guaranteed miss.
--- tests
Five, in `tests/test_lmcache_offload_connector.py`, on a `CacheEngineKey`
fixture rather than the full lmcache stub (these are about two keys being equal
or not, which the real dataclass answers by value): a layout change makes a
different key; the layout does not collapse distinct hashes; the key survives a
restart; a state key cannot equal a raw-hash KV key; a wrong-sized hit is a miss
with its reference discharged; an object that will not report a size is still
loaded.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3911 passed, zero failures. black clean;
ruff 36, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 3c. Adds `_KimiMLAGDNCommon.page_unit_views(unit_ids)` -- the tensor-view counterpart of #2045's `_page_unit_regions`. No behaviour changes; nothing calls it until Phase 3d. --- what changed - `page_unit_views(unit_ids) -> list[torch.Tensor]`, one view per (unit, MLA row), unit major and row minor. - Seven tests in `tests/test_kda_checkpoint_slot_copy.py`, on the harness #2045 already built for `_page_unit_bases`. --- why a second way to name the same bytes `_page_unit_regions` hands the Triton descriptor raw int64 addresses, which is what a D2D copy the GPU issues wants. The LMCache staging packer takes `list[torch.Tensor]` -- `fused_pack_chunk_major` is already a fully parameterised gather over `segment_ptrs[] + segment_block_bytes[]`, so it needs no kernel change, only its segments named the way it names them. This is the ONLY new seam the tier needs. A load writes the Active Slot directly and reuses `state_entry_views`, so the asymmetry is deliberate: the save gathers from units, the load scatters to a slot, and the two compose because #2045's `plan_segmented_copy` intersects two *ordered byte streams* -- the slot's ranges and the units' regions are two cuts of the same logical image, so a blob gathered in one order is byte-identical to one gathered in the other. --- the two traps, both asserted rather than assumed *Order is the contract.* Unit major, row minor -- the same ravel as `_page_unit_bases`, which is the order `_checkpoint_copy_plan` builds the destination stream in. The packer indexes segments positionally, so this order IS the blob layout. Self-consistency would be enough for one build; what makes it a contract across builds is `layout_id` in the key (Phase 3b), so a build that reordered cannot read another's blob. *`block_ratio`.* `unit_ids` carries **logical** block ids while `kv_cache` is shaped in **physical** ones, and K3's ratio is 128. Flattening the two block axes together and indexing by `unit * block_size` is the conversion -- the same arithmetic `_page_unit_bases` does in addresses, so the two cannot drift into disagreeing about which bytes a unit owns. The range check is against the *logical* count for the same reason: the physical count is `ratio` times larger in rows, so checking the wrong axis admits ids that read past the end. Its test computes the bound from the tensor rather than from a constant, because taking the harness's label would let the very confusion under test pass unnoticed. `_page_unit_regions` is called first for its side effect: the contiguity and granularity checks live there, and they have to run before any view is handed out rather than after. --- verification The unit tests skip on a non-GPU runner (the module imports aiter at load), so the geometry was checked directly by extracting the four methods and comparing `page_unit_views` against `_page_unit_bases` byte for byte: 9 segments for 3 units x 3 rows, identical `data_ptr()`s, lengths equal to `_page_unit_stream_sizes`, every view contiguous, no two units sharing a byte, an out-of-range id refused, and a granularity mismatch refused before any view is produced. pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors that fail on #2045 too) -> 3911 passed, zero failures. black clean; ruff 36, unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… evicted
Phase 3d, the last of Phase 3. This is what turns the tier back on for saves:
a checkpoint that reaches READY is offered to the CPU tier, and the whole
engine->worker->engine round trip is wired. Loads are still off (Phase 4).
--- what changed
`page_unit_checkpoint.py`
- `_offload_ready`: checkpoints nominated at the READY transition. NOT pinned.
- `take_offload_stores(max_inflight)`: hands `(hash, unit_ids)` over and takes
the pin at that moment, up to the cap.
- `settle_offload_store(hash)` / `reclaim_stale_offload_pins(timeout_s)` /
`offload_pins_reclaimed`, exposed on the coordinator and reported in
`checkpoint_fates`.
- `PageUnitCheckpointStore(..., offload_sink=)` -- False nominates nothing.
`block_manager.py`
- `take_state_stores` / `settle_state_store(hash, ok)` /
`reclaim_stale_state_store_pins`. The switch (`kv_connector_hosts_state_tier`)
moves above the coordinator's construction, which now needs it.
`scheduler.py`
- `_publish_state_stores()`, beside `_publish_state_loads()` and on the same
two call sites plus the idle path.
- `_offload_max_pending_saves()`, reading the KV leg's own
`OFFLOAD_MAX_PENDING_SAVES`.
- The report drain settles the pin on both outcomes and runs the reconciler.
`kimi_k3/connector.py` -- `enqueue_state_stores`, `meta.state_stores`,
`_start_state_stores`, and `_state_store_failed_locally` for stores a worker
with no tier cannot even attempt.
`kimi_k3/state_tier.py` -- `submit_store(h, unit_ids)`, no `ready_event`.
`kimi_k3/state_object.py` -- `put(h, unit_ids)` gathers `page_unit_views`;
`get(h, slot)` still scatters into `state_entry_views`; `puts_refused`.
`metadata.py` / `types.py` -- `state_stores` declared as work.
--- why READY, and why the pin is not taken there
Three candidate trigger points and only one works. `begin_store` is too early:
the scatter has not ridden a batch, so the units do not hold the image yet.
`_evict` / `unindex` are too late -- they fire when the pool wants those units
*now*, so a multi-millisecond D2H started there holds 127 of the most-wanted
blocks at the worst possible moment. READY is when the bytes first exist and
when the record is least wanted: it has just entered the LRU at the cold end.
The pin is a separate question, and getting it wrong broke a real invariant --
`test_a_ready_unpinned_checkpoint_is_available_to_live_kv` caught the first
draft. A pin lives in this process while the copy runs in the worker, so it
spans several scheduler passes. Pinning at READY would make EVERY checkpoint
un-evictable for that window, which is exactly what #2045's admission argument
forbids. So READY only *nominates*: a nominee stays unpinned and spendable, and
`take_offload_stores` pins only the few actually in flight. A checkpoint the
pool needed more than the tier did simply loses its copy, which is the right
price for never making the pool wait on the CPU.
The cap is `OFFLOAD_MAX_PENDING_SAVES`, shared with the KV leg rather than given
a knob of its own: a KV save and a state store both hold bytes out of the same
pool while they run, so the depth is one question about one resource, and two
numbers would let an operator raise one and unknowingly double the other.
--- write volume, which is the objection this trigger invites
Storing every checkpoint rather than only evicted ones is more traffic, and it
is affordable: per request state adds 53.6 MiB on top of ~404 MB of KV, **13%**
on a path already in flight (~12 MB/s against the KV leg's ~90 MB/s over a
3600s run). What it buys is the trigger being at the cheapest possible moment
and dedup for free -- `begin_store` already refuses a hash it holds or has
pending, so one hash is offered once.
--- unindex during a copy does not cancel it
`unindex` means the boundary's KV block was spent, so the image is unreachable
in HBM from here on. That is precisely when the CPU copy is worth having, so
the pin holds the units to the end of the D2H and `_release_record` runs after.
The CPU copy outliving the HBM one is the entire point of the tier.
--- the lost report, and why 20 lines is enough
`reclaim_stale_offload_pins` takes the same window as
`_reconcile_stalled_deferred_saves`, derived from LMCache's own pin timeout for
the same reason. Recovery is total: a leaked pin breaks no `BlockPool`
invariant and leaves no half-released state, so zeroing the count restores the
record exactly -- which is why this needs a counter and a timer rather than any
repair logic. `offload_pins_reclaimed` reaches `checkpoint_fates`, because
otherwise the symptom is a pool that has quietly shrunk.
--- still off
Loads. Nothing calls `_request_state_load` for K3 yet: `_attach_state_slots`
returns from its PAGE branch before reaching `_tier_can_serve`, and the
`begin_restore` raise above it has to become an issue-a-load branch first
(Phase 4). `StateOffloadIndex.hashes` now fills, so the tier is half live: it
writes, and nothing reads yet.
--- tests
11 new in `tests/test_page_unit_checkpoint.py` -- nomination does not pin, the
hand-off does, a nominee stays spendable and may be lost, no sink nominates
nothing, offered once, the cap makes the rest wait rather than drop, a waiting
nominee stays evictable, both outcomes release, a report for a hash never sent
is a no-op, `unindex` mid-copy holds the units to the end, and a lost report is
reclaimed with the record left unbroken. Two more in
`tests/test_lmcache_offload_connector.py`: stores drain exactly once, and a
step whose only work is a store still reaches the worker.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3924 passed, zero failures. black clean;
ruff 36, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…raised
Phase 4a, and it has to be first: with nothing else changed this commit is
behaviour-preserving, but without it the first request that actually uses the
CPU tier takes the engine down.
--- what changed
`BlockManager._attach_state_slots`, PAGE branch. Was:
if hit_hash != -1 and not begin_restore(hit_hash, seq.state_slot):
release_many(seq.state_slots); seq.state_slots = []
raise RuntimeError("gated PAGE checkpoint disappeared before state attach")
Now: cold start returns early; `begin_restore` success returns as before; a
miss falls through to `_request_state_load`, and only if that also declines is
the boundary disowned (`return False`, which `allocate` turns into
`num_cached_tokens = 0`). New counter `state_gate_lost_boundary`, reported in
`checkpoint_funnel`.
--- why the raise could not stay
It asserted that the gate and the HBM store agree. That held while
`can_allocate` only ever accepted boundaries the HBM index carried. Phase 4b
gives the CPU tier a vote, so a hash the gate accepted may live only there --
and every such request would hit this raise. Turning it into a load branch has
to land before the vote does, not after, or the two commits are only safe as a
pair.
--- why disowning is reachable rather than defensive
Two ways to arrive with neither tier able to produce it:
* the tier's index is optimistic by construction -- `hashes` means "was
stored once", never "is still there", because LMCache's own LRU can drop
bytes at any time;
* an HBM checkpoint can be unindexed between `can_allocate` and `allocate`
by another seq's `_fresh_block` in the same pass.
So this is a normal path with a counter, not an assertion. A large fraction of
`joint_boundaries` landing here would mean the gate is accepting boundaries
that do not survive to attach.
--- the slots are kept, unlike the old abort
The old code released the slots because it was aborting. Disowning is not an
abort: the request is about to recompute its whole prefix, and it writes that
state into these very slots. Releasing them would hand the next request a
buffer this one is still filling.
`test_missing_gated_checkpoint_releases_the_new_slot_and_raises` becomes
`test_a_gated_boundary_neither_tier_has_is_disowned_not_raised`, and now pins
all three: returns False, keeps its slots, counts one.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…m it
Phase 4b. This is the commit that makes the tier reachable: until now K3 wrote
53.6 MiB per checkpoint to LMCache and nothing ever read it.
--- what changed
`page_unit_checkpoint.py`
- `PagedStateCheckpointCoordinator.offload` + `attach_offload(index)`.
- `resumable_hit` now asks `_reachable(h)` instead of `store.contains(h)`:
HBM first, then `offload.hashes`.
- The constructor's `offload_sink: bool` is gone -- `attach_offload` sets the
vote and the sink together (see below).
`block_manager.py`
- Attaches the index to the coordinator once both exist. The coordinator is
built from `state_runtime.checkpoint_spec` and the index from the connector
config, so they are constructed in the wrong order for a ctor argument.
`kimi_k3/connector.py`
- Refuses to build the tier when `image_bytes != entry_bytes`.
--- why the vote is the whole feature
`can_allocate` walks the HBM prefix and asks every `Pool.STATE` class "the
rightmost boundary <= X that you accept". With the coordinator answering only
from `store.contains`, a hash whose image went to LMCache is invisible: the
walk stops short, nothing ever asks for it back, and the store path is pure
cost. `_reachable` is where the tier gets to say "I have that one".
It stops at the first boundary it accepts, scanning right to left, so a hash
accepted here that nothing can deliver does not cost a wasted lookup -- it
costs the whole walk-back, hiding every shorter checkpoint still resident in
HBM. That is why the disown path (Phase 4a) had to land first.
No preference rule between the tiers is needed: both are keyed by the same
content hash, so the scan takes the rightmost boundary wherever it lives, and
`_attach_state_slots` tries `begin_restore` before `_request_state_load`
regardless. A resident image never pays a park.
--- why the vote is optimistic, deliberately
`hashes` means "was stored once", never "is still there" -- LMCache's own LRU
can drop bytes under it at any moment. A false positive costs one park plus a
recompute, and retracts itself: `fail_load` calls `forget(h)`, so the next
request over that prefix is not sent down the same hole. Being certain instead
would mean a synchronous cross-process lookup on the admission path.
--- why attaching is one call, not two fields
Half-attached is worse than off in both directions. A vote with no sink accepts
hashes nothing will ever store, so every accepting request parks and
recomputes. A sink with no vote pins 127 units per checkpoint for a transfer
nobody will read. `attach_offload` sets both, so they cannot diverge.
--- the load's destination is the Active Slot
Not 127 freshly reserved units. The two are interchangeable because #2045's
copy plan intersects two *ordered byte streams* -- the slot's ranges and the
units' regions are two cuts of one image -- so a blob gathered in unit order
unpacks correctly in slot order. Writing the slot directly means no
`ensure_free_units` on the admission path, no refusal branch, no units pinned
for the length of an H2D, and no cap needed on concurrent loads.
It does mean the store reads `image_bytes` and the load writes a whole slot, so
the connector now refuses to build a tier where those differ, at the one point
both numbers are in scope. They are equal for K3; a model where they are not
would silently truncate the store or over-read the load.
--- tests
Six in `tests/test_page_unit_checkpoint.py`: a tier-only hash is accepted, no
tier means no acceptance, the scan still takes the rightmost, attaching turns
on both halves together, and the optimistic half retracts on `forget`.
Three in `tests/test_state_checkpoint.py`, end to end through
`_attach_state_slots`: a tier-only boundary becomes a queued load with no
restore op and the slot as its destination; a boundary in both is served from
HBM with no park; a tier that declines disowns and counts.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> zero failures. black clean; ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ason to exist
Phase 5, the last one. `_joint_kv_boundary` refused every PAGE sequence; it now
serves only PAGE. With this the tier is reachable end to end.
--- what changed
`block_manager.py`
- The gate inverts:
- if not seq.has_per_req_cache or self.paged_state_checkpoints is not None:
- return self._no_joint("not_hybrid")
+ if not seq.has_per_req_cache or self.paged_state_checkpoints is None:
+ return self._no_joint("no_paged_checkpoints")
- The hbm/tier split reads the coordinator instead of the slot pool.
- `joint_boundaries_hbm` / `_tier` become `state_hbm` / `state_tier`.
`page_unit_checkpoint.py` -- `contains(h)` delegation, so the split does not
reach into `.store`.
`scheduler.py` / `llm_engine.py` -- the log line and the Prometheus tuple follow
the rename.
--- why the old refusal was right, and why it stopped being right
It read `is not None` and refused, because a K3 checkpoint was an Active Slot
that the tier spilled out of `StateSlotPool` -- a PAGE image lived in the KV
pool, which the KV connector already offloads on its own, so there was no
second boundary to agree on.
#2045 moved K3's image into the KV pool and with it the entire reason the joint
path exists. HBM now enforces `state ⊆ KV`: `_record_evicted` unindexes a
checkpoint the instant its boundary block is spent, so a checkpoint can no
longer outlive its KV there. **When LMCache hands the KV back, nothing hands
the state back unless the two are fetched together.** Leaving this refusal in
place makes the tier dead weight -- it writes 53.6 MiB per checkpoint and
nothing ever reads it -- and no other counter in the system would say so.
--- why the counters were renamed rather than left alone
`joint_boundaries_hbm/_tier` described the boundary as a whole, which was true
when the KV leg had no tier of its own to be served from. It has one now
(`_decide_load_after_alloc` decides it separately), so the old names read as if
they covered both legs. `state_hbm` / `state_tier` say what they measure.
The split also means something new. It used to say "the state pool is too small
for this concurrency". Under #2045 checkpoints share the paged pool with the KV
write stream, so a ratio dominated by `state_tier` says **the paged pool** is
too small -- checkpoints are being squeezed out by the KV they now live beside.
That diagnosis did not exist before.
`state_tier` is also the only counter here that cannot be non-zero with the CPU
tier switched off, which makes it the honest test of "did this feature run".
No passing unit test can produce it in production.
--- tests
Three in `tests/test_state_checkpoint.py`, on a harness that had to reproduce
the shape the joint path is for: publish a prompt, place ONE rung at an
interior boundary (a checkpoint under the last block is one no scan can look
up, since `can_allocate` matches over `range(n_hash_blocks - 1)`), then evict a
KV block below it so the walk stops short. A PAGE seq gets a joint boundary at
the rung with real work for the KV leg; the split reports `hbm` while the
checkpoint is resident and `tier` once it is not; no tier means no boundary.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors that fail on #2045 too) -> 3935 passed, zero failures. black clean;
ruff unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…res`
Found by the first end-to-end run of the K3 CPU state tier on this branch: the
tier reported healthy on both sides and stored exactly nothing. 94 checkpoints
were nominated, handed to `_publish_state_stores`, refused, and had their PAGE
units released again.
--- the symptom
Server log, 94 times over a 6-minute agentic replay at concurrency 8:
state offload: LMCacheOffloadConnectorScheduler did not carry 1 state
store(s); releasing their units now.
with everything else looking correct:
lmcache_offload: worker family=kimi_k3
kimi_k3 offload: state tier up, entry=55.39 MiB rank=0, sharing the
paged-KV CPU pool, layout=kda-paged-state-v1:layers=69:...
state checkpoints: checkpoints_kept=112 ... offload_pins_reclaimed=0
So the worker half built the tier, the coordinator nominated checkpoints at
READY, and `take_offload_stores` pinned and handed them over -- and the
scheduler half dropped every one.
--- the cause
`LMCacheOffloadConnectorScheduler` is a shell that owns no behaviour: it holds
`self._impl = _build_scheduler(config)` and forwards each method by hand.
`7b6b58e0` added `enqueue_state_stores` to `KimiK3OffloadScheduler`
(kimi_k3/connector.py:352) but did not add the matching forwarder to the shell,
so the method existed on the implementation and was invisible from outside:
$ git grep -c enqueue_state_stores atom/kv_transfer/offload/connector.py
0
`Scheduler._publish_state_stores` probes with
`getattr(self.kv_connector, "enqueue_state_stores", None)`, and
`self.kv_connector` is the shell, so the probe returned None unconditionally
and the store path took its "nothing will carry these" branch on every pass.
This is not a race or a config error -- it could never have worked.
The load leg was unaffected only by luck: `enqueue_state_loads` and
`take_state_reports` were both given forwarders when they were added. Loads
still moved no bytes, because with nothing ever stored `offload.hashes` stayed
empty and the tier had nothing to vote for.
--- the fix
Add the missing forwarder, in the same shape as its `enqueue_state_loads`
neighbour so the two cannot drift again.
--- the second change, and why it is in this commit
The warning that reported the failure named a cause that was not the one:
The tier's index should not have been installed against this connector.
That reads as a configuration mistake, and it sent the first pass of this
diagnosis at `kv_connector_hosts_state_tier` and the layout selector -- both of
which were behaving correctly. The class name printed is the *shell*, which
tells you nothing about whether `_impl` supports the call. Both warnings now
name the two possible causes and say which class is being reported.
--- verification
Static: the probe is `getattr` on a method the shell does not define, so the
refusal is unconditional; adding the forwarder is the whole of the repair, and
it is additive -- no existing path changes.
Runtime: re-running the same workload with this commit applied, the
`did not carry` warning no longer appears.
Lines conform to black's 88-column limit; black and ruff are not installed in
the benchmark image, so formatting was checked by inspection against the
surrounding style rather than by running them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…that holds none Three observability defects, found while trying to judge whether the K3 CPU state tier was working. Together they meant the tier could store nothing at all and no number anywhere would say so -- which is exactly what happened in 8a5364a, where 94 refused stores produced one warning line and four counters that all read zero for unrelated reasons. --- 1. the store leg had no counters at all `StateOffloadIndex.stats()` reported `loads_attempted / completed / failed` and `indexed`. On a K3 run the load counters stay zero until something has been stored, so the only store-side signal was `indexed` growing -- and `indexed` is a set size that also grows for reasons unrelated to this pass. Added `stores_attempted` (handed to the connector, counted in `take_state_stores`), `stores_completed` / `stores_failed` (split in `settle_state_store` by the worker's report), and `stores_refused`. `stores_refused` is deliberately apart from `stores_failed`, and it is the one that matters most: failed means the worker tried and could not, refused means nobody tried. The shell-forwarding bug fixed in 8a5364a was a pure `refused` condition, and with this counter it would have been one glance at the periodic line instead of a hunt through the connector factory. The worker's own `puts_refused` needs no separate channel: a refused `put` already returns False and settles as `ok=False`, so it lands in `stores_failed`. --- 2. `[Pool Pressure]` read the pool that holds no checkpoints `BlockManager.pool_pressure()` folded in `self.state.checkpoint_fates()`. Under #2045 a K3 checkpoint is a PAGE image the coordinator owns and the slot pool holds none, so the line printed [Checkpoint Fates] kept: 0, dropped: 0, evicted: 0, orphaned: 0 on the same run whose other line said `checkpoints_kept=112`. Two counters for one fact, disagreeing in one log, with the wrong one printed in the section a reader goes to for pool pressure. Now reads `self._state_checkpoint_cache`, which is already the coordinator when there is one and the slot pool otherwise -- the same object every other checkpoint question is asked of. `occupancy()` stays on the slot pool, which is the thing that has slots. --- 3. the tier's counters vanished when K3's checkpoints moved to PAGE `StateGroupPool.checkpoint_fates` used to fold the tier's stats in under a `state_offload_` prefix. `PagedStateCheckpointCoordinator.checkpoint_fates` was written without that, so once #2045 moved K3's checkpoints into the coordinator the whole `state_offload_*` family disappeared from the logs -- a silent regression, since nothing errors when a key stops being emitted. Restored on the coordinator. The fold is `getattr`-guarded because `attach_offload` accepts anything answering `hashes`, and the tests attach a double with no counters; a missing `stats` means "nothing to fold", not an error. (`test_a_tier_that_declines_disowns_rather_than_parking` caught this on the first draft.) --- also added: `state_offload_store_backlog` A gauge rather than a counter: `len(_offload_ready)`, the nominations waiting for a slot under `OFFLOAD_MAX_PENDING_SAVES`. That queue is unbounded and drained oldest-first, so checkpoints reaching READY faster than the tier drains them shows up as a value that climbs and stays high, with fresh nominations queueing behind stale ones. There is no other way to see it -- the stale entries are skipped cheaply on the way out, so nothing else registers the backlog. --- how to read the new line state_offload_stores_refused > 0 -- wiring fault, the connector never took it stores_attempted - completed - failed -- in flight stores_completed vs checkpoints_kept -- how much of what HBM keeps the tier got loads_failed / loads_attempted -- the index's false-positive rate store_backlog climbing -- READY outruns the drain Verified: pytest tests/test_page_unit_checkpoint.py tests/test_state_checkpoint.py tests/test_lmcache_offload_connector.py tests/test_block_manager.py -> 466 passed. Lines conform to black's 88-column limit; black and ruff are absent from the benchmark image, so formatting was checked by inspection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every state store on this branch has failed since the store path was wired. Found by the first run where stores were actually carried (8a5364a): 3,904 tracebacks, all identical, and not one checkpoint reached LMCache. --- the symptom Traceback (most recent call last): File "kimi_k3/state_tier.py", line 109, in _do_store stored = bool(self.codec.put(h, unit_ids)) File "kimi_k3/state_object.py", line 141, in put self._staged.pack(self._backend.page_unit_views(unit_ids), obj) File "kimi_k3/staging.py", line 175, in pack from atom.kv_transfer.offload.triton_kv_staging import fused_pack_chunk_major ModuleNotFoundError: No module named 'atom.kv_transfer.offload.triton_kv_staging' state offload: store of hash 10785477046942485201 failed 3,904 of them over a 50-minute concurrency-16 agentic replay -- one per store attempt, on every rank. `stores_failed` would have counted all of them, and `indexed` never moved. --- cause 1: the module is one package deeper `fused_pack_chunk_major` lives in `atom.kv_transfer.offload.dense.triton_kv_staging`. The dense codec, which has used it all along, imports it from there (`dense/kv_byte_codec.py:127`). The K3 stager's import dropped the `.dense`. Nothing caught it because the import is function-local, inside `pack`, so it is only attempted when a store actually runs -- and until 8a5364a no store ever reached the worker. The unit tests do not reach it either: they exercise the codec against doubles that never call the real packer. --- cause 2: a failed `put` leaked its allocation, once per failure obj = self._allocate(self.entry_bytes) # ref_count = 1 self._staged.pack(...) # raises self._storage.batched_put([key], [obj]) # never reached `batched_put` discharges the reference it is handed (`StorageManager.batched_put` ends in `memory_obj.ref_count_down()`), so the success path correctly owes nothing -- but only if it is reached. With `pack` throwing, the allocation was stranded at ref_count=1 and LMCache reported it much later: MemoryObj at ... is being garbage collected with ref_count=1, pin_count=0. This indicates ref_count_down()/unpin() was not called 3,768 such warnings against 3,904 failed stores: one leak per failure. Left alone this shrinks the shared CPU pool by one 53.6 MiB entry per failed store until nothing can be allocated at all -- so the second-order effect of the import bug was a slow starvation of the pool the KV leg shares. `get` already guarded its own reference with `finally` for exactly this reason; `put` did not. It does now, on the exception path only, matching what the DSV4 codec does (`hybrid/dsv4/codec.py:1293`). --- why one commit The leak is only reachable through a throwing `pack`, and the import was the only thing throwing. Fixing the import alone would hide the leak rather than remove it, and the next exception from that line -- an out-of-memory in the staging buffer, say -- would strand allocations again with no traceback to explain the pool shrinking. --- verification python -c "from atom.kv_transfer.offload.dense.triton_kv_staging import fused_pack_chunk_major" -> OK pytest tests/test_lmcache_offload_connector.py tests/test_state_checkpoint.py tests/test_page_unit_checkpoint.py -> 412 passed The import is still function-local, so this remains untested by anything that does not run a real store. A unit test that calls `StagedTransfer.pack` against real CUDA tensors would have caught it; that is worth adding and is not in this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot reach
Fix-then-sweep over `dfab0fa0` and `8a5364ab`. Both of those repaired one
instance of a bug that had two, and neither instance the fix reached is covered
by any test.
--- 1. `unpack` imports the same module that does not exist
`dfab0fa0` corrected `StagedTransfer.pack`'s import of
`fused_pack_chunk_major` (the module is one package deeper, under `.dense`).
`unpack` imports `fused_unpack_chunk_major` from the same wrong path, eleven
lines further down, and was left alone.
It has never fired because no state load has ever run -- with nothing stored,
`offload.hashes` stayed empty and the tier had nothing to vote for. The first
successful store makes the first load reachable, and it would have raised
`ModuleNotFoundError` on that load, on every rank.
Both kernels live in `atom.kv_transfer.offload.dense.triton_kv_staging`;
verified by importing both.
--- 2. `chunk_size` is not on the delegating shell either
`8a5364ab` added the missing `enqueue_state_stores` forwarder. Sweeping every
member `Scheduler` reads off `self.kv_connector` against what
`LMCacheOffloadConnectorScheduler` defines turns up one more: `chunk_size`.
scheduler.py:2284 seq.offload_kv_chunk_tokens = int(
getattr(self.kv_connector, "chunk_size", 0) or 0)
`self.kv_connector` is the shell, the shell has no `chunk_size`, so this has
always stamped 0. It is latent rather than live: `_joint_kv_boundary` reads
`self._joint_chunk_tokens` first and only falls back to this, and that primary
comes from `BlockManager.__init__` reading the LMCache config directly. But the
fallback exists precisely for the case where that read failed -- and it was
dead, so a build that could not read the config at startup would refuse every
joint boundary with `no_chunk_size` and never say why.
Forwarded as a property, with the reason in its docstring.
--- 3. the class of bug, closed rather than fixed twice
Both are the same shape: the shell forwards by hand, so a member added to an
implementation is invisible from outside, and every reader is a
`getattr(..., default)` that does not raise -- it takes the default forever.
Two instances in two days says the next one is a matter of time.
`test_every_member_the_scheduler_reads_is_reachable_through_the_shell` scans
`scheduler.py`'s source for `getattr(self.kv_connector, "x")`,
`self.kv_connector.x`, and `_connector_flag("x")`, and asserts every name is
reachable on the shell class. It reads the source rather than a hand-kept list,
because a hand-kept list is the same failure one level up. The scan asserts it
found something, so a regex that stops matching fails loudly instead of passing
vacuously.
Falsified: renaming the new `chunk_size` property makes it fail and name the
missing member.
Verified: pytest tests/ (less tests/plugin and the two msgpack/aiter collection
errors) -> 3936 passed, zero failures. black clean; ruff 36, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…it were image
Fourth and last blocker in the chain that kept the K3 CPU tier from storing
anything. With the previous three fixed, a 25-minute concurrency-16 replay
still ended at
state_offload_stores_attempted=462
state_offload_stores_completed=0
state_offload_stores_failed=450
state_offload_indexed=0
and 3,696 identical tracebacks:
File "kimi_k3/state_object.py", line 149, in put
self._staged.pack(self._backend.page_unit_views(unit_ids), obj)
File "kimi_k3/staging.py", line 182, in pack
dst_tensor = self.memory_tensor(dst, nbytes)
ValueError: ATOM LMCache connector: MemoryObj tensor is too small for
59867136 bytes; got 58079232
--- the geometry
From the run's own startup line:
unit_bytes=2138112 image_bytes=58079232 units_per_checkpoint=28
ceil(58079232 / 2138112) = ceil(27.163) = 28
28 x 2138112 = 59867136 <- what page_unit_views gathered
58079232 <- what put() allocated
difference = 1787904 <- padding in the 28th unit
An image occupies whole units, and the last one is 83.6% padding here. Nothing
owns those bytes.
--- why only the CPU side hit it
The HBM path never had the problem because it already passes the image size
down: `_checkpoint_copy_plan` calls
plan_segmented_copy(src_sizes, self._page_unit_stream_sizes(units),
spec.image_bytes)
so the D2D copy writes exactly `image_bytes` into the unit stream and leaves
the tail untouched. The image is, by construction, the LEADING `image_bytes`
of that stream. `page_unit_views` was written as the tensor-view counterpart of
the same addressing but did not carry the third argument across, so it named
whole units to a consumer that sizes its destination at `entry_bytes` -- which
for K3 is exactly `image_bytes`.
The connector's `image_bytes != entry_bytes` guard cannot catch this: both are
58079232 and agree. The quantity that disagrees is a third one,
`sum(page_unit_views)`, which no check compared against either.
--- the fix
Trim the gathered stream to `image_bytes`, dropping whole views past the
budget and slicing the one that straddles it.
The slice is byte-exact rather than view-aligned because the image does not end
on a view boundary: 58079232 leaves 350208 B of the last unit in use, which is
20.96 of its rows. That view is reinterpreted as `uint8` and sliced; the packer
sizes segments as `numel * element_size`, so a uint8 segment costs it nothing.
The load side needs no counterpart: it scatters into `state_entry_views`, which
sums to `entry_bytes` exactly, so both ends of the round trip are now the same
58079232 bytes.
A budget left unspent raises rather than storing a short blob -- that would be
a real geometry disagreement, and a truncated image read back as valid is the
one outcome worth crashing over.
--- two notes on shape
Inline rather than a helper method: the existing tests drive `page_unit_views`
with a `types.SimpleNamespace` stand-in for `self`, and a second attribute would
have to be stubbed in every one of them. For the same reason the spec is read
with `getattr(spec, "image_bytes", 0)` -- a fork build carries no spec at all,
and there the whole-unit stream is the right answer.
--- verification
pytest tests/test_kda_checkpoint_slot_copy.py
tests/test_lmcache_offload_connector.py
tests/test_state_checkpoint.py
tests/test_page_unit_checkpoint.py -> 441 passed
Also confirmed from the same run that the previous commit's leak guard works:
`MemoryObj ... garbage collected with ref_count=1` went from 3,768 occurrences
to 0, even though every store still failed.
None of the four bugs in this chain was reachable by the unit tests, because
all of them live past the first line that only a real store executes. A test
that packs real CUDA tensors through `StagedTransfer` would have caught three
of the four; it is still worth adding.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
About half of every K3 state store failed. Measured over a concurrency-16
agentic replay with the four earlier blockers fixed:
state_offload_stores_attempted=46
state_offload_stores_completed=19
state_offload_stores_failed=27 <- 59%
with 296 identical tracebacks:
File "kimi_k3/state_object.py", line 153, in put
self._storage.batched_put([self.key(h)], [obj])
^^^^^^^^^^^
File "kimi_k3/state_object.py", line 107, in key
digest.update(int(h).to_bytes(8, "little", signed=True))
OverflowError: int too big to convert
state offload: store of hash 12983297547834104159 failed
--- cause 1: the hash is unsigned and the key said otherwise
12983297547834104159 > 2**63-1, so `to_bytes(8, signed=True)` cannot hold it.
An ATOM block hash is `xxhash.xxh64().intdigest()`, which spans the whole
0..2**64-1, and `BlockManager.compute_hash` chains one into the next with
`prefix.to_bytes(8, "little")` -- unsigned, no keyword. The state key was the
only place in the chain that read the same value as signed, so it rejected
every hash with the top bit set. That is half the key space, and the observed
59% is that with sampling noise.
Nothing downstream noticed because the failure is inside a `try` in
`_do_store`, which counts a failed store and moves on. Before the counters
added in a804561 it produced no number at all.
--- cause 2: the raise stranded a MemoryObj, in a window the last guard missed
`dfab0fa0` wrapped `pack` so a throwing gather would discharge the allocation.
`key(h)` is evaluated in `batched_put`'s argument list, which is after the
allocation and before `batched_put` can take ownership -- outside that guard.
So each of the 296 overflows also leaked a 53.6 MiB entry, and the LMCache
warning reappeared next to the traceback:
MemoryObj at 136839168 is being garbage collected with ref_count=1
Fixed by computing the key *before* allocating. `key` is pure, so hoisting it
removes the window rather than guarding it, and the `except` now covers
`batched_put` itself as well -- matching what the DSV4 codec does.
--- what this unblocks
The same `key` is used by `get`, so roughly half of all loads would have missed
in a way indistinguishable from an LRU eviction -- `fail_load` would then
`forget` a hash whose bytes were never stored under a name it could find, and
the index would decay for a reason nothing reported.
Verified: pytest over the state and offload suites -> 442 passed. The overflow
itself is confirmed directly:
>>> (12983297547834104159).to_bytes(8, "little", signed=True)
OverflowError: int too big to convert
>>> (12983297547834104159).to_bytes(8, "little", signed=False).hex()
'5f19dacf54f02db4'
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`961f80d6` fixed `page_unit_views` gathering a checkpoint's padding, and the
suite it ran stayed green before and after — because no test reaches the trim
at all.
The addressing tests build their stub spec as
checkpoint_spec=SimpleNamespace(page_unit_bytes=...)
with no `image_bytes`, so `budget` is 0 and every one of them takes the
`return views` early path. The trim shipped untested, which is how a bug that
failed 3,696 stores in a row got there in the first place.
--- what is pinned
Seven cases in `TestPageUnitViewsStopAtTheImage`, on the harness the addressing
tests already use (`build` gains optional `image_bytes` and `dtype`; both stay
off by default so the existing tests keep exercising whole units):
* the gathered stream is exactly `image_bytes` — the invariant `put`'s
allocation depends on, and the one that was broken;
* a whole view past the budget is dropped, not truncated to zero length;
* an image that lands on a view boundary is not sliced at all;
* **the kept bytes are the leading bytes and nothing else** — trimming
truncates, it does not reorder. The blob is read back by scattering into
the slot in the same order, so a byte that moved here would land in the
wrong layer there with nothing raised;
* a multi-byte dtype is sliced by BYTES, not elements — the straddling view
is reinterpreted as `uint8` first, and on bf16 an element-wise slice would
keep twice what it should. The production cache is 1 byte per element, so
only a test with a wider dtype can hold this;
* a spec with no image size keeps whole units (the fork build, which has no
spec to trim against);
* units that cannot cover the image raise rather than storing a short blob.
--- verification
`tests/test_kda_checkpoint_slot_copy.py` skips wholesale on a non-GPU runner
(the module imports aiter at load), so the seven cases were additionally run by
extracting `page_unit_views` and its three neighbours out of the source and
driving them directly. All seven hold, including the bf16 one.
pytest tests/ (less tests/plugin and the two msgpack/aiter collection errors)
-> 3936 passed, zero failures. black clean; ruff 36, unchanged.
--- one note on `put`'s exception shape, checked not changed
`2c5c3457` moved `batched_put` inside the `try` whose `except` calls
`ref_count_down`. That is safe only if `batched_put` takes ownership just on
success — and it is the pattern the DSV4 codec already uses
(`hybrid/dsv4/codec.py:1284`, "StorageManager owns the MemoryObj after a
successful batched_put"). Consistent with the repo rather than a new
assumption, so left alone.
The window between `_allocate` and `batched_put` is now fully covered:
`page_unit_views` can raise (the geometry check above) and is inside the try;
`key(h)` is computed before the allocation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two admission-path voters -- `PagedStateCheckpointCoordinator. _reachable` and `BlockManager._tier_can_serve` -- tested `h in offload.hashes`, dropping the `can_load` half that `request_load` itself enforces. A store-only role (`kv_producer`) populates `hashes` from its own stores but cannot load them back, so it voted a tier hit it would then refuse: the right-to-left resumable scan stopped at the tier rung, skipped a still-resident HBM rung, and `request_load` returned False -- the boundary disowned and the HBM checkpoint forfeited to a full recompute. Factor the membership+capability predicate onto the index as `StateOffloadIndex.could_serve(h) = can_load and h in hashes` (the guard of `request_load` minus its per-request in-flight check) and route both voters and `request_load` through it, so all three test exactly the same capability and cannot drift -- fix-then-sweep across the three read sites. Test doubles for the index gain the mirror method.
|
Thanks for the round-6 pass — every finding was actionable. Fixed each as its own commit on top of the review head (
Detail per finding below. 1 — Arm joint loads from state ids, not
|
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a new offload layout and cross-cutting scheduler/worker/lifecycle changes where subtle correctness regressions (silent wrong output or hangs) require careful human validation beyond automated review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
tests/test_multi_connector.py:64
FakeSchedSubdefaultshas_state_tierto True wheneveroffload_methods=True, but the realLMCacheOffloadConnectorScheduler.has_state_tieris only True for thekimi_k3layout (dense/hybrid offload shells should report False). This makes the test double behave unlike production and can mask routing bugs where state calls should be rejected when no tier is present.
Consider defaulting has_state_tier to False (while still defining the attribute) and having individual tests opt-in with has_state_tier=True only when they explicitly model the K3 state tier.
- Files reviewed: 51/51 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a new cross-process joint KV+state offload path with broad engine/connector lifecycle changes where subtle correctness or liveness regressions are hard to rule out automatically.
Review details
- Files reviewed: 51/51 changed files
- Comments generated: 1
- Review effort level: Lite
| self._state_source_released = set() | ||
| return released | ||
|
|
||
| def take_state_reports(self) -> tuple[set[int], set[int]]: |
|
Sixth pass, re-reviewed from scratch at Round 5: seven of eight fixed, and fixed well. Not repeated below — 1. A multimodal prompt that spans a checkpoint rung livelocks the scheduler
if atomic_prefill and chunk < seq.num_tokens - seq.num_cached_tokens:
self.block_manager.deallocate(seq)
self.waiting.appendleft(seq)
breakThe re-assert is correct in intent, but it differs from the pre-allocation guard in the one way that matters:
And the adjuster that shortened the chunk is deterministic. target = bm.checkpoint_cut(seq, start, start + chunk)
if target:
chunk = target - start
So for The pre-alloc guard at One correction to how this was reported to me: the two adjusters are not equivalent here. The new test cannot see it. Options, roughly in order of how much they preserve the invariant: suppress the rung cut when 2. The
|
…rantee holds Finding 3: start_load_kv() called self._arm_joint_loads(metadata) OUTSIDE the try/finally that guarantees super().start_load_kv() runs. _arm_joint_loads builds park state and can raise; if it did, the finally-super() KV-leg fallback was skipped and the KV load was silently dropped. Move the arm inside the try next to the two state helpers so every state-side helper is covered by the finally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… state-owning model
Finding 2: select_offload_layout() mapped an explicit offload_layout override and
returned it BEFORE inspecting the model, so an override could (a) bypass the GDN
refusal and (b) downgrade a kimi_linear (kimi_k3) checkpoint to a layout that owns
no tier for its per-request KDA recurrent state -> a KV prefix restored over stale
state = silent wrong output.
Fix: always compute the natural layout via the extracted _layout_from_model()
first (which still refuses GDN), then, if an override is present, refuse mapping a
state-owning natural layout (_STATE_OWNING_LAYOUTS = {"kimi_k3"}) down to a
non-state layout. dense<->hybrid remains a legitimate operator choice (namespace
separation), so the guard is deliberately narrow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g a checkpoint-rung cut Finding 1: _finalize_prefill_chunk() always aligned the prefill chunk to a state-checkpoint rung via checkpoint_cut(). For a multimodal prompt (taken as one atomic unit) whose length is off the rung grid, the deterministic cut kept shaving the chunk to the same sub-rung length every pass -> the request could never make prefill progress (livelock). Fix: when seq.multimodal_data is present, return the whole chunk before checkpoint_cut(). A checkpoint is only ever kept where a forward ends on a rung, so an off-grid whole prompt simply keeps no mid-prompt checkpoint; forks do not arise on a fresh multimodal request, so no correctness is lost. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… re-admission cannot drop a slot Still-open 2: _orphan_load_slots mapped req id -> a single (slot, ts) tuple. If a request was preempted and re-admitted while a state load was still in flight, the second park overwrote the first entry and leaked the first slot; settle/abandon also released the wrong (only) slot. Fix: store a list of (slot, ts) per id, append on each park, and release FIFO via _release_oldest_orphan_slot() (matching worker report order). settle_state_load and abandon_state_load release the oldest; reconcile_orphan_load_slots expires stale entries per-slot by age while keeping fresh ones. The entry is deleted when its list empties. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for the round-6 pass — the three new findings were all real. Fixed one commit per finding on the merged tip (
Finding 1 — multimodal prefill livelock —
|
There was a problem hiding this comment.
🔵 Needs a closer look
A couple of incorrect public type annotations for state-tier store reports (reported as int rather than StateStoreOperationId) should be corrected to avoid misuse and tooling confusion in a correctness-critical area.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
atom/kv_transfer/offload/_offload_common.py:293
StateOffloadFace.take_state_reports()is annotated as returningtuple[set[int], set[int]], but the scheduler consumes these asStateStoreOperationIdobjects (passed toBlockManager.settle_state_store(op, ...)). The current annotation is misleading and can cause incorrect assumptions in type-aware tooling.
atom/kv_transfer/offload/hybrid/kimi_k3/connector.py:898
take_state_reports()drainsSTATE_INDEX_CHANNELcompletions whoseoperation_idis aStateStoreOperationId, not aninthash. The current return typetuple[set[int], set[int]]is incorrect and conflicts with how the scheduler callsBlockManager.settle_state_store(op, ...).
def take_state_reports(self) -> tuple[set[int], set[int]]:
"""Drain this step's tier store reports for the engine-side index.
Both real tier failures and no-tier worker failures land in
`_state_index_failed` via STATE_INDEX_CHANNEL, so there is a single
source of truth to drain.
"""
indexed = self._state_indexed
failed = self._state_index_failed
self._state_indexed = set()
self._state_index_failed = set()
return indexed, failed
- Files reviewed: 51/51 changed files
- Comments generated: 0 new
- Review effort level: Lite
Reviewing what PR #2053's six rounds left behind turned up a gap in this branch rather than in that one. The shell-forwarding sweep pins a `must_be_seen` set so that a later refactor hiding one of the scheduler's `getattr(self.kv_connector, ...)` reads shrinks `probed` loudly instead of silently. Fusing the load leg removed `enqueue_state_loads` from that surface but ADDED `take_missed_state_hashes` (scheduler.py:3070) -- so the guarded surface did not shrink, and the one member whose read is new was the one member not anchored. The regex matches it today; nothing was pinning that it keeps doing so. Also drops a comment naming `handed_off`, a flag removed before this branch began. The reasoning it carried is the non-obvious half -- `batched_put` discharges the caller's reference in its terminal tail loop, so there is no adopted-then-raised window -- so it stays, rewritten as the constraint it is ("do not reintroduce a flag that skips the down") rather than as a story about a symbol no longer in the tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Answering a fair challenge: if the debt was architecture-induced, why is none of it redeemable? Auditing what #2053 changed in PRE-EXISTING code -- rather than what its fixes added -- found one keep of mine that does not survive. `MultiConnector._state_tier` was written twice and read never. The two-tier refusal reads `_state_tier` off the SUBS; the four state forwarders route through `_state_tier_sub()`, a different method that also probes the subs. The comment justified the mirror as making a probe resolve, but no such probe exists: `start_load_kv` fans each sub its own metadata, so the K3 sub uses its own field. Its two tests asserted the mirror against a rationale -- "the K3 store/load path reads `_state_tier` off whatever connector the forward context holds; under `multi` that is this object" -- which is simply not what the code does. The method's real and only product is the refusal, so it is renamed to say so. The refusal, its ValueError and its test are untouched: a config listing two offload subs would leave a hash reported indexed by one tier and fetched from another, and that is worth a loud failure at model load. Also drops `lmc-state-load` from a docstring listing the threads `close` must join -- that executor went with the fusion. 4252 passed (4254 - 2 tests whose subject no longer exists). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sweeping #2053's ~4,200 added test lines for tests whose subject the fusion removed found none -- but it found the opposite: a test that looked like coverage and was not. `test_success_and_failure_release_identically` was `for _ in range(2):` around five identical statements, both iterations taking the success path. It could not have tested a failure: `PageUnitCheckpointStore.settle_offload_store` takes no outcome argument, so at that level there is no distinction to make. Renamed to what it actually pins, and it now points at where the distinction really lives. That distinction is `BlockManager.settle_state_store(op, ok=..., attempted=...)`, whose docstring states it precisely -- and `ok=False` was called by NO test in the repo. `grep "ok=False\|attempted=False" tests/` returned nothing; all three call sites passed `ok=True`. So the failure legs of a state store were untested: that a failed store still releases its units but must NOT index its hash (voting for a hash whose bytes never landed sends the next request over that prefix to a `get` that must miss), and that a REFUSED store is not additionally counted as failed and backs out its optimistic `stores_attempted` bump. Both now have tests. Also renames `TestStateLoadsAndStoresRunInSeparateLanes`: there are no lanes since the fusion made the load synchronous on the KV task's own thread, and the class docstring now says the property is structural rather than scheduled. 4254 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewing what PR #2053's six rounds left behind turned up a gap in this branch rather than in that one. The shell-forwarding sweep pins a `must_be_seen` set so that a later refactor hiding one of the scheduler's `getattr(self.kv_connector, ...)` reads shrinks `probed` loudly instead of silently. Fusing the load leg removed `enqueue_state_loads` from that surface but ADDED `take_missed_state_hashes` (scheduler.py:3070) -- so the guarded surface did not shrink, and the one member whose read is new was the one member not anchored. The regex matches it today; nothing was pinning that it keeps doing so. Also drops a comment naming `handed_off`, a flag removed before this branch began. The reasoning it carried is the non-obvious half -- `batched_put` discharges the caller's reference in its terminal tail loop, so there is no adopted-then-raised window -- so it stays, rewritten as the constraint it is ("do not reintroduce a flag that skips the down") rather than as a story about a symbol no longer in the tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Answering a fair challenge: if the debt was architecture-induced, why is none of it redeemable? Auditing what #2053 changed in PRE-EXISTING code -- rather than what its fixes added -- found one keep of mine that does not survive. `MultiConnector._state_tier` was written twice and read never. The two-tier refusal reads `_state_tier` off the SUBS; the four state forwarders route through `_state_tier_sub()`, a different method that also probes the subs. The comment justified the mirror as making a probe resolve, but no such probe exists: `start_load_kv` fans each sub its own metadata, so the K3 sub uses its own field. Its two tests asserted the mirror against a rationale -- "the K3 store/load path reads `_state_tier` off whatever connector the forward context holds; under `multi` that is this object" -- which is simply not what the code does. The method's real and only product is the refusal, so it is renamed to say so. The refusal, its ValueError and its test are untouched: a config listing two offload subs would leave a hash reported indexed by one tier and fetched from another, and that is worth a loud failure at model load. Also drops `lmc-state-load` from a docstring listing the threads `close` must join -- that executor went with the fusion. 4252 passed (4254 - 2 tests whose subject no longer exists). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sweeping #2053's ~4,200 added test lines for tests whose subject the fusion removed found none -- but it found the opposite: a test that looked like coverage and was not. `test_success_and_failure_release_identically` was `for _ in range(2):` around five identical statements, both iterations taking the success path. It could not have tested a failure: `PageUnitCheckpointStore.settle_offload_store` takes no outcome argument, so at that level there is no distinction to make. Renamed to what it actually pins, and it now points at where the distinction really lives. That distinction is `BlockManager.settle_state_store(op, ok=..., attempted=...)`, whose docstring states it precisely -- and `ok=False` was called by NO test in the repo. `grep "ok=False\|attempted=False" tests/` returned nothing; all three call sites passed `ok=True`. So the failure legs of a state store were untested: that a failed store still releases its units but must NOT index its hash (voting for a hash whose bytes never landed sends the next request over that prefix to a `get` that must miss), and that a REFUSED store is not additionally counted as failed and backs out its optimistic `stores_attempted` bump. Both now have tests. Also renames `TestStateLoadsAndStoresRunInSeparateLanes`: there are no lanes since the fusion made the load synchronous on the KV task's own thread, and the class docstring now says the property is structural rather than scheduled. 4254 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewing what PR #2053's six rounds left behind turned up a gap in this branch rather than in that one. The shell-forwarding sweep pins a `must_be_seen` set so that a later refactor hiding one of the scheduler's `getattr(self.kv_connector, ...)` reads shrinks `probed` loudly instead of silently. Fusing the load leg removed `enqueue_state_loads` from that surface but ADDED `take_missed_state_hashes` (scheduler.py:3070) -- so the guarded surface did not shrink, and the one member whose read is new was the one member not anchored. The regex matches it today; nothing was pinning that it keeps doing so. Also drops a comment naming `handed_off`, a flag removed before this branch began. The reasoning it carried is the non-obvious half -- `batched_put` discharges the caller's reference in its terminal tail loop, so there is no adopted-then-raised window -- so it stays, rewritten as the constraint it is ("do not reintroduce a flag that skips the down") rather than as a story about a symbol no longer in the tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
[Kimi-K3][LMCache] LMCache offload for Kimi-K3
Adds a CPU offload tier for K3's KDA recurrent state, alongside the KV tier
lmcache_offloadalready provides. A prefix whose KV came back from LMCache isuseless without the recurrent state that goes with it, so the two are fetched
together or not at all.
Builds on #2045, which turned a K3 checkpoint from a whole Active Slot into a
PAGE image living in ordinary KV blocks. That is what makes this tier cheap:
the source of a save is a set of reserved blocks that nobody can take away,
so nothing has to be rescued anywhere before the D2H.
Design
A third layout, inheriting
denserather than written beside it.K3's paged KV is ordinary dense MLA — block-addressed, sliceable, moved by
chunk — so
KimiK3OffloadConnector/KimiK3OffloadSchedulerextendDenseOffloadConnector/DenseOffloadSchedulerand add one extra leg for the69 layers of KDA state. (DSV4 is written in parallel instead because its PAGE
leg carries an indexer and has a different shape.) Every override wraps a dense
call rather than replacing it; the KV leg's behaviour is unchanged.
Layout selection is config-only (
select_offload_layout), because thescheduler process has no model.
Both legs save when the bytes land in HBM, into one pool.
READYshould_defer_freeNo second engine and no
OFFLOAD_STATE_CPU_SIZE. A request writes its KVchunks and its one state object inside the same prefill window, so both enter
the LRU together and cool at the same rate — state is retired alongside its own
KV, which is what we want, since a boundary whose KV is gone is worthless. Two
pools would buy two eviction policies that must drift, while a joint boundary
needs both legs to survive together.
Loads pick one boundary for both legs. The ceiling comes from KV, the
landing point from state; the KV leg fetches into blocks the request already
holds, and the state leg's H2D writes the request's Active Slot directly — no
units are reserved for it.
Everything is asynchronous.
submit()returns immediately; the Tritongather runs on its own stream, the D2H on another, and the engine advances on
the worker's report. Nothing waits on the compute stream.
Why the boundary has to be joint
KDA state at position
Bis the compressed history of exactly[0, B). Raisethe KV-loaded length past
Band the linear layers never see the tokens inbetween: wrong output, no exception. So the two legs cannot be decided
separately.
Example
A request whose prompt is 12 hash blocks,
A B C D E F G H I J K L(
--block-size 128,LMCACHE_CHUNK_SIZE 256). BlockChas been evicted fromHBM; LMCache still holds the KV for
A..Kand the state checkpoint filed atH.can_allocatewalks the HBM prefix and stops at the first miss,C.Two blocks matched.
B, so the request may claim 0 tokens ascached.
_joint_kv_boundary:gives 10; capped by
n_hash_blocks - 1(prefill must forward one block toproduce logits) → 10;
continued past where HBM stopped, which is what makes a boundary that
exists only in LMCache addressable at all;
H→B = 1024;B= 1024;allocatepoints the block table atA B— free, since those blocks sitbelow where the forward will start — and leaves
num_cached_tokensat 0.H.[256, 1024)= C D E F G H. The already-residentA Bisnot re-fetched; doing so would move bytes the GPU holds and leave an
unindexed second copy of them in HBM.
_claim_after_load→ 1024, the state boundary, not the transfer end.num_cached_tokens = 1024andthe forward computes
I J K L.Three positions, all different, and each is needed: 0 is what may be called
cached, 256 is what the block table may point at, 1024 is where both
legs are aimed. Claiming past the first is safe only because a joint boundary
exists — those blocks are below the forward's start, so nobody writes them.
Transferring past
Bcosts one chunk the forward immediately rewrites;claiming past
Bis silent wrong output.Notes for reviewers
state_pool.pyandsub_pool_spec.pyhave an empty diff against main.The tier does not reach into the slot pool at all.
layout_id.CacheEngineKeyhas no field saying whatan entry is, and KV and state keys come from different hash functions into one
integer space — now sharing one pool. Folding the layout in makes the two
spaces disjoint by construction, and separately stops a build that changed the
state geometry from reading another's image back as valid.
READYbut pinned only when handed to theworker, bounded by
OFFLOAD_MAX_PENDING_SAVES. Pinning atREADYwould makeevery checkpoint un-evictable for a window that spans several scheduler
passes, breaking the admission rule that a READY unpinned checkpoint counts
as space available to live KV.
(
fused_pack_chunk_major), unmodified.OFFLOAD_STATE,OFFLOAD_KV_FOR_HYBRID,OFFLOAD_SAVE_MAX_INFLIGHT,OFFLOAD_SAVE_STALL_S,OFFLOAD_STATE_MIN_LOAD_TOKENS,OFFLOAD_STATE_CPU_SIZE,OFFLOAD_STATE_STAGING_GROUPS. A resumable prefix needs both legs, so theconnector being configured is the only switch.
Verification
CON16

CON32